Asynchronous Distributed Neural Network Training using Alternating Direction Method of Multipliers

Since the first appearance of a large-scale dataset [4] and powerful computational resources such as GPUs, Convolutional Neural Networks(CNN) became the essential machine learning algorithm for image classification, detection, and many application. As the popularity of CNN increases, the size of CNN increased as well[10, 13]. (For instance, AlexNet has more than 60 milion parameters.) The performance of such large networks improves steadily yet training these networks takes longer. In this paper, we propose a novel asynchronous distributed neural network optimization using Alternating Direction Method of Multipliers(ADMM)[6]. Unlike previous works on distributed optimization for neural network training [1, 5, 3, 12] which rely soley on the primal optimization, we formulate the problem into a global consensus optimization and distribute the neural network training in a principled fashion. We evaluate our framework on CIFAR10 dataset and analyze the effect of hyperparameters introduced in ADMM.